Technical Field
The present disclosure generally relates to the field of electronic data processing and file collection and management systems. More specifically, and without limitation, the exemplary embodiments described herein relate to computer-implemented systems and methods for distributed file collection and processing in, for example, high latency, distributed networks.
Background
Online content systems and related services are used for handling the distribution and placement of electronic content on the Internet. Such content may include articles, videos, and other content, including online advertisements. In the field of online advertising, for example, computerized systems and services are provided for managing and placing advertisements for goods and services on web pages viewed by users throughout the world. Online advertising systems collect information about advertisements from ad creators and profile information from users and process this information in order to determine how to deliver ads to users who are likely to be interested in particular goods or services. Similar techniques are also used for other forms of electronic content placed on web pages.
Several different parties may be involved in an online content system. For example, with reference again to the online advertising field, such parties may include advertisers, publishers, users (i.e., consumers of goods and services), and online advertising service providers. Multiple parties and systems are also involved when dealing with the distribution of other online content for the Internet. In most cases, the relevant parties are scattered geographically throughout the world. As a result, the systems and hardware components involved in collecting and processing pertinent data may likewise be scattered across multiple regions or continents. For example, an online advertising system may comprise ad servers, bid servers, processing and/or collection centers, and consumer systems spread across multiple regions or continents. As the number of entities and users involved in these systems and the amount of data exchanged by these parties grows, the disparate locations of these computerized components can create data latencies that hinder the ability to serve the creators and recipients of relevant data.
Current online content systems also suffer from downtime due to static system configuration and other factors. For example, the types of data to be collected from an ad server or a bid server may be configured in a static file, such that the addition of any new file types requires elements of the system to be shut down. User data may also be configured in a static file, such that the addition of new recipients of pertinent data may require that certain system elements be shut down. As online advertising systems and other content systems grow to accommodate additional content creators and recipients, and change to handle new types of data, the downtime caused due to static configuration may lead to unacceptably frequent service interruptions.
As a result, there is a need for improved systems and methods for distributed file collection and processing that can adapt and elastically expand and contract with minimal configuration and downtime. There is also a need for systems and methods to automatically discover, register, and/or remove data and react to changes in data types provided by data producers. Moreover, there is a need for systems and methods that can dynamically react to the addition of new consumers or changing needs of existing data consumers.